Augmented reality maps

ABSTRACT

A handheld communication device can capture and display a real-time video stream. The handheld communication device detects a geographic position and camera direction of the handheld communication device. A route is identified from the geographic position of the handheld communication device to a point of interest. The captured video stream is visually augmented with an indicator indicating a direction to travel to the point of interest. The indicator is overlaid on the captured real-time video stream.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/341,915 filed Nov. 2, 2016, which is a continuation of U.S. patentapplication Ser. No. 12/705,558 filed on Feb. 12, 2010, the contents ofwhich are incorporated by reference in their entirety for all purposes.

FIELD

The following relates to searching for nearby points of interest, andmore particularly to displaying information related to nearby points ofinterest overlaid onto a video feed of a surrounding area.

BACKGROUND

Augmented reality systems supplement reality, in the form of a capturedimage or video stream, with additional information. In many cases, suchsystems take advantage of a portable electronic device's imaging anddisplay capabilities and combine a video feed with data describingobjects in the video. In some examples, the data describing the objectsin the video can be the result of a search for nearby points ofinterest.

For example, a user visiting a foreign city can point a handheldcommunication device and capture a video stream of a particular view. Auser can also enter a search term, such as museums. The system can thenaugment the captured video stream with search term result informationrelated to nearby museums that are within the view of the video stream.This allows a user to supplement their view of reality with additionalinformation available from search engines.

However, if a user desires to visit one of the museums, the user mustswitch applications, or at a minimum, switch out of an augmented realityview to learn directions to the museum. However, such systems can failto orient a user's with a poor sense of direction and force the user tocorrelate the directions with objects in reality. Such a transition isnot always as easy as it might seem. For example, an instruction thatdirects a user to go north on Main St. assumes that the user can discernwhich direction is north. Further, in some instances, street signs mightbe missing or indecipherable, making it difficult for the user to findthe directed route.

SUMMARY

Such challenges can be overcome using the present technology. Therefore,a method and system for displaying augmented reality maps are disclosed.By interpreting the data describing the surrounding areas, the devicecan determine what objects are presently being viewed on the display.The device can further overlay information regarding the presentlyviewed objects, thus enhancing reality. In some embodiments, the devicecan also display search results overlaid onto the displayed video feed.Search results need not be actually viewable by a user in real life.Instead, search results can also include more-distant objects.

The user can interact with the display using an input device such as atouch screen. Using the input device, the user can select from amongobjects represented on the screen, including the search results.

In one form of interaction, a device can receive an input from the userrequesting directions from a present location to a selected searchresult. Directions can be overlaid onto the presently displayed videofeed, thus showing a course and upcoming turns. As the user andassociated device progress along a route, the overlaid directions canautomatically update to show the updated path.

In some embodiments the display can also include indicator graphics topoint the user in a proper direction. For example, if the user is facingsouth but a route requires the user to progress north, “no route” wouldbe displayed in the display because the user would be looking to thesouth but the route would be behind him or her. In such instances, anindicator can point the user in the proper direction to find the route.

In some embodiments, multiple display views can be presented based onthe orientation of the device. For example, when the device is held atan angle with respect to the ground of 45 degrees to 180 degrees, thedisplay view can present the augmented reality embodiments describedherein. However, when the device is held at an angle less than 45degrees, an illustrated or schematic view can be represented. In suchembodiments, when the device is held at an angle with respect to theground of less than 45 degrees, the device is likely pointed at theground, where few objects of interest are likely to be represented inthe displayed video. In such instances, a different map view is morelikely to be useful. It should be appreciated that precise range of tiltcan be adjusted according the actual environment or user preferences.

In practice, a user points a handheld communication device to captureand display a real-time video stream of a view. The handheldcommunication device detects a geographic position, camera direction,and tilt of the image capture device. The user sends a search request toa server for nearby points of interest. The handheld communicationdevice receives search results based on the search request, geographicposition, camera direction, and tilt of the handheld communicationdevice. The handheld communication device visually augments the capturedvideo stream with data related to each point of interest. The user thenselects a point of interest to visit. The handheld communication devicevisually augments the captured video stream with a directional map to aselected point of interest in response to the user input.

A method of augmenting a video stream of a device's present surroundingwith navigational information is disclosed. The user can instruct thedevice to initiate a live video feed using an onboard camera and displaythe captured video images on a display. By polling a Global PositioningSystem (GPS) device, a digital compass, and optionally, anaccelerometer, location, camera direction, and orientation informationcan be determined. By using the location, camera direction, andorientation information, the device can request data describing thesurrounding areas and the objects therein. In some embodiments, thisdata includes map vector data. The can be requested from an onboardmemory or a server. The data describing surrounding areas can further berequested in conjunction with a search request. The search request canalso include a request for information about nearby places of interest.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary visually augmented captured image withdata related to a search for points of interest;

FIG. 2 illustrates the results of a field-of-view and point-of-interestsearch;

FIG. 3 illustrates an exemplary captured image visually augmented with aroute to a selected point of interest;

FIG. 4 is a flow chart illustrating an exemplary method of preparing anddisplaying an augmented reality map;

FIG. 5 is a schematic illustration of an exemplary system embodiment;and

FIG. 6 is a schematic illustration of an exemplary system embodiment.

DESCRIPTION

The technology described herein visually augments a captured image orvideo stream with data for points of interest related to search termsentered by the user. The technology also visually augments the capturedimage or video stream with a directional map to a selected point ofinterest.

FIG. 1 is a screenshot illustrating an augmented reality embodiment asdescribed herein. As illustrated, a handheld communication device hascaptured an image 102 of the northwest corner of the intersection ofDolores Stand 17th St. using its image-capturing device and displayedthe image on its display. In this way, the display can function as aviewfinder. As illustrated, the captured image 102 has been augmentedwith information corresponding to points of interest 104, 106 and streetlabels 110, 112.

FIG. 1 illustrates a captured and presented image 102 using an imagecapture device, i.e., the camera of a smart phone, which is but one typeof handheld communication device to which the present disclosure can beapplied. In this illustrated embodiment, the user has entered a searchterm “parks” in search bar 108 to conduct a search for nearby parks,i.e., a specific type of point of interest. Using map data thatdescribes the area surrounding the present location of the device andthe points of interest located in the surrounding area, the deviceaugments the displayed image with additional information. In thisinstance, the smart phone or handheld communication device displayspoints of interest described by the data that are displayed in theviewfinder (such as Dolores St. 110 and 17th St. 112) or within a fieldof view and range from the geographic position of the device but thatare obstructed by other in-screen objects, e.g., Golden Gate Park 104and Buena Vista Park 106. While other parks might also be nearby, theyare not shown because they fall outside the field of view of the device.However, the user could locate these parks by panning the device aroundthe intersection, in which case those parks would appear on the screen.

In the captured image 102, the handheld communication device augmentsthe captured image with bubbles showing the relative geographic positionof “Golden Gate Park” 104 and “Buena Vista Park” 106 within the capturedimage 102. This allows the user to determine a general direction to apoint of interest. A user can then select a point of interest, e.g., byselecting the “Buena Vista Park” 106 point of interest informationbubble, e.g., by touching the point of interest information bubble witha finger or stylus if the smart phone employs a touch screen. In otherimplementations, a cursor and mouse can be used to select a desiredpoint of interest.

Points of interest can be any map feature, but most often a point ofinterest can be a map feature that identified as result of a search fora category of such map features. For example, a point of interest can bea park when a user searches for nearby parks. Likewise a point ofinterest can be places, buildings, structures, even friends that can belocated on a map, when the point of interest is searched for. In someinstances a point of interest is not necessarily identified as a resultof a search. A point of interest can also be a map feature that isidentified by the present system because it can be viewed in thecaptured image. In short, a point of interest can be any map feature forwhich the user has an interest.

FIG. 2 illustrates search results for point of interest results fornearby parks based on geographic position and also illustrates how arange and field of view correspond to the results displayed in theviewfinder. A handheld communication device captures a video stream ofthe view as shown in FIG. 1. The handheld communication device detectsthe geographic position, camera direction, and tilt of the handheldcommunication device.

The geographic position of the handheld communication device can bedetermined using GPS coordinates or using triangulation methods usingcell phone towers. In yet another example, a blend of GPS coordinatesand triangulation information can be used to determine the position ofthe device.

The camera direction is a direction relative to a planet's magneticfield (i.e., Earth's magnetic field) in which the camera is pointing.The camera direction can be considered a direction that can beidentified using a compass, such as a digital compass. The cameradirection can be used to identify the direction in which the camera ispointing as it acquires an image to be augmented using the presenttechnology.

The tilt direction is a direction that determines the direction in whicheither the camera device or display device is pointing relative to ahorizontal or vertical axis. The tilt direction can most commonly bedetermined using an accelerometer.

The user can enter a search request for nearby points of interest basedon a search term. In this example, upon entry by the user of a searchfor nearby “Parks” the handheld communication device sends a request fordata related to nearby parks to a map database.

Either the request itself, or the database being queried can determine arelevant range from within which search results must be encompassed.Upon receipt of the request, the database will return search results forpoints of interest related to the search term that are also within adefined radius of the handheld communication device as illustrated inFIG. 2. As shown in this example, the server returned points of interest“Golden Gate Park” 208, “Buena Vista Park” 206, “Midtown TerracePlayground” 210, and “Mission Dolores Park” 212. The handheldcommunication device determines that of the point-of-interest searchresults, only “Golden Gate Park” 208 and “Buena Vista Park” 206 arewithin the field of view of the handheld communication device. Thepoint- of-interest results “Golden Gate Park” 208 and “Buena Vista Park”206 are displayed with their relative spatial relationship to thehandheld communication device. In the example shown in FIG. 2, thecamera direction of the handheld communication device is northwest.

A field of view can be determined using a digital compass to inform thedevice of the camera direction in which the camera is facing or,alternatively, the user could enter in a heading. As explained above, inFIGS. 1 and 2, the camera is facing northwest and its theoretical lineof sight is represented as 214 in FIG. 2. Any search results that are tobe displayed on the viewfinder must be within a certain angle of line214. For example, a camera on a handheld communication device might onlybe able to display range of view encompassing 30 degrees. In such aninstance, a given display would represent those items encompassed within15 degrees in each direction from the center of the field of view. Thisconcept is illustrated in FIG. 2 wherein 214 illustrates the center ofthe field of view and angles θ₁ 216=θ₂ 218 and they represent anglesfrom the center of the field of view to the outer limits of the field ofview. A distance from the device's geographic location can also be usedto define a field of view. As discussed above, a distance or range canbe defined by the device in its request for search results or by thedatabase serving the request. Only search results encompassed in thisfield of view will be displayed on the display.

In some embodiments, a device can also use an accelerometer to informthe device of what objects are displayed in its viewfinder. For example,if the device is in a hilly location, the accelerometer can tell thedevice that it is pointing downhill. In another example, the device candetermine that, due to the topography surrounding its present location(described by map data) an object viewed at a certain angle from thehorizon must be a neighboring hill or mountain peak in the distance. Inyet another example, an angle from a horizon can indicate that the useris viewing a multiple story building having places of interest inmultiple stories of the building. An accelerometer can inform the deviceof the angle at which the device is pointed.

FIG. 3 illustrates a captured image that has been visually augmentedwith route data to a selected point of interest. In this example, a userhas selected the “Buena Vista Park” point of interest and, in response,the smart phone has visually augmented the captured image 302 with adirectional map 310 to the selected point of interest, i.e., “BuenaVista Park”. The route shows a direction 312 that the user must travelon Dolores St. to begin travelling to reach “Buena Vista Park.” Thedirectional map 310 further indicates a turn 314 that the user musttake, i.e., a turn left onto Duboce Ave. from Dolores St. In theillustrated example, the map is shown overlaid onto Dolores St.

The route 310 guides the user with complete navigation illustrations toreach “Buena Vista Park,” including any required turns. In someembodiments, the route can be represented as a schematic map, i.e., asimplified map that includes only relevant information for the user inan easy-to-read format.

A schematic map can be thought of as similar to a subway map one wouldsee on a subway train. While the subway track itself might wind andturn, a typical subway map represents the subway route as a mostlystraight line. Further, the subway map often does not have anyparticular scale and frequently shows every destination approximatelyevenly dispersed along the route. Thus, a schematic map as discussedbelow is one that does not adhere to geographic “reality,” but ratherrepresents map features in a schematic fashion by illustratingdirections as a route made of one or more roads, trails, or ways thatcan be represented as substantially straight lines instead of by theiractual shapes (which would be represented in a non-schematic map byadhering to geographic reality). The schematic map can also be devoid ofuniform scale. Thus, in some parts of the map, such as an area of themap representing a destination, such area can be “distorted” somewhat toclearly illustrate important details, while map areas that representportions of a route where there are no turns or other significantfeatures can be very condensed. In short, the map can be a schematic ofthe real world that can provide a simple and clear representation thatis sufficient to aid the user in guidance or orientation withoutdisplaying unnecessary map features or detail that could otherwiseclutter a small display space.

FIG. 4 is a flow chart illustrating an exemplary method of preparing anddisplaying an augmented reality map. As shown at block 402, the methodincludes capturing and displaying a video stream on a handheldcommunication device. Although described here in reference to a videostream, another embodiment of the disclosed technology includescapturing and displaying a single still image or a series of stillImages.

As shown at block 404, the method includes detecting geographicposition, camera direction, and/or tilt of the handheld communicationdevice. This allows the device to determine features, such as streets,buildings, points of interest, etc., that are within a field of view forthe captured video stream.

As shown at block 406, the method includes sending a request for nearbypoints of interest based on one or more search terms. For example, theuser can search for nearby hotels, parks, or restaurants. The requestcan be sent to a database located on a server that is separate from thehandheld communication device and communicate via a wireless protocol.In another embodiment, the database can be stored locally on the deviceand the search request remains internal (sometimes termed “onboard” thedevice) to the handheld communication device.

In block 408, the method includes receiving nearby points of interest inresponse to the request. The server can filter point of interest resultsin one example. In this example, if the number of returned points ofinterest exceeds a set threshold, the server can filter the results toonly return a fixed number of the best results. Various algorithms canbe employed to filter points of interest to a desired number for visualaugmentation of a captured video stream. In another embodiment, thehandheld communication device can filter point-of-interest resultsreceived from the server for optimal display on a handheld communicationdevice.

In block 410, the handheld communication device visually augments thecaptured video stream with data related to each point of interest. Asshown in FIG. 2, the handheld communication device can visually augmenta captured video stream with a bubble for each point of interest withinthe field of view for the handheld communication device. The handheldcommunication device determines which points of interest are within itsfield of view by analyzing the geographic position, camera direction,and/or tilt of the handheld communication device in concert with theknown geographic position of the returned points of interest.

In block 412, the handheld communication device visually augments thecaptured video stream with a directional map to a selected point ofinterest in response to the user input. For example, as described inconnection with FIG. 3, the smart phone now visually augments thecaptured image 302 with a directional map 310 to the selected point ofinterest in response to the user input. The user input can be aselection of a displayed point of interest to indicate that the userwishes to view navigation data for reaching the selected point ofinterest.

In some embodiments, the display can also include indicator graphics topoint the user in a proper direction. For example, if the user is facingsouth but a route requires the user to progress north, “no route” wouldbe shown in the display because the route would be behind him or her. Insuch instances, an indicator can point the user in the proper directionto find the displayed route.

In some embodiments, multiple display views can be presented based onthe orientation of the device. For example, when the device is held atan angle with respect to the ground of 45 degrees to 180 degrees, thedisplay view can present the augmented reality embodiments describedherein. However, when the device is held at an angle less than 45degrees, an illustrated or schematic view can be presented. In suchembodiments, when the device is held at an angle with respect to theground of less than 45 degrees, the device is likely pointed at theground, where few objects of interest are likely to be represented inthe displayed video. In such instances, a different map view than theaugmented reality map is more likely to be useful. It should beappreciated that precise range of tilt can be adjusted according to theactual environment or user preferences.

FIG. 5 illustrates a computer system 500 used to execute the describedmethod and generate and display augmented reality maps. Computer system500 is an example of computer hardware, software, and firmware that canbe used to implement the disclosures above. System 500 includes aprocessor 520, which is representative of any number of physicallyand/or logically distinct resources capable of executing software,firmware, and hardware configured to perform identified computations.Processor 520 communicates with a chipset 522 that can control input toand output from processor 520. In this example, chipset 522 outputsinformation to display 540 and can read and write information tonon-volatile storage 560, which can include magnetic media and solidstate media, for example. Chipset 522 also can read data from and writedata to RAM 570. A bridge 535 for interfacing with a variety of userinterface components can be provided for interfacing with chipset 522.Such user interface components can include a keyboard 536, a microphone537, touch-detection-and-processing circuitry 538, a pointing devicesuch as a mouse 539, and so on. In general, inputs to system 500 cancome from any of a variety of machine-generated and/or human-generatedsources.

Chipset 522 also can interface with one or more data network interfaces525 that can have different physical interfaces 517. Such data networkinterfaces can include interfaces for wired and wireless local areanetworks, for broadband wireless networks, as well as personal areanetworks. Some applications of the methods for generating and displayingand using the augmented reality user interface disclosed herein caninclude receiving data over physical interface 517 or be generated bythe machine itself by processor 520 analyzing data stored in memory 560or 570. Further, the machine can receive inputs from the user viadevices keyboard 536, microphone 537, touch device 538, and pointingdevice 539 and execute appropriate functions, such as browsing functionsby interpreting these inputs using processor 520.

While FIG. 5 illustrates an example of a common system architecture, itshould also be appreciated that other system architectures are known andcan be used with the present technology. For example, systems whereinmost or all of the components described within FIG. 5 can be joined to abus, or the peripherals could write to a common shared memory that isconnected to a processor or a bus can be used. Other hardwarearchitectures are possible and such are considered to be within thescope of the present technology.

FIG. 6 illustrates an exemplary system embodiment. A server 602 is inelectronic communication with a handheld communication device 618 havingfunctional components such as a processor 620, memory 622, graphicsaccelerator 624, accelerometer 626, communications interface 628,compass 630, GPS 632, display 634, input device 636, and camera 638.None of the devices are limited to the illustrated components. Thecomponents may be hardware, software, or a combination of both.

In some embodiments, the server can be separate from the handheldcommunication device. The server and handheld communication device cancommunicate wirelessly, over a wired-connection, or through a mixture ofwireless and wired connections. The handheld communication device cancommunicate with the server over a TCP/IP connection. In anotherembodiment, the handheld communication device can be directly connectedto the server. In another embodiment, the handheld communication devicecan also act as a server and store the points of interest locally.

In some embodiments, instructions are input to the handheld electronicdevice 618 through an input device 636 that instructs the processor 620to execute functions in an augmented reality application. One potentialinstruction can be to generate an augmented reality map of traveldirections to a point of interest. In that case, the processor 620instructs the camera 638 to begin feeding video images to the display634. In some embodiments, video images recorded by the camera are firstsent to graphics accelerator 624 for processing before the images aredisplayed. In some embodiments, the processer can be the graphicsaccelerator. The image can be first drawn in memory 622 or, ifavailable, memory directly associated with the graphics accelerator 624.

The processor 620 can also receive location and orientation informationfrom devices such as a GPS device 632, communications interface 628,digital compass 630 and accelerometer 626. The GPS device can determineGPS coordinates by receiving signals from Global Positioning System(GPS) satellites and can communicate them to the processor. Likewise,the processor can determine the location of the device throughtriangulation techniques using signals received by the communicationsinterface 628. The processor can determine the orientation of the deviceby receiving directional information from the digital compass 630 andtilt information from the accelerometer.

The processor can also direct the communications interface to send arequest to the server 602 for map data corresponding to the areasurrounding the geographical location of the device. In someembodiments, the processor can receive signals from the input device,which can be interpreted by the processor to be a search request for mapdata including features of interest.

The processor can interpret the location and orientation data receivedfrom the accelerometer 626, compass 630, or GPS 632 to determine thedirection in which the camera 638 is facing. Using this information, theprocessor can further correlate the location and orientation data withthe map data and the video images to identify objects recorded by thecamera 638 and displayed on the display 634.

The processor can receive other inputs via the input device 636 such asan input that can be interpreted as a selection of a point of interestdisplayed on the display 634 and a request for directions. The processor620 can further interpret the map data to generate and display a routeover the displayed image for guiding the user to a destination (selectedpoint of interest).

As the user follows the specified direction to the selected points ofinterest, the processor can continue to receive updated location anddirectional information and video input and update the overlaid route.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer-readable media. Such instructions comprise, forexample, instructions and data which cause or otherwise configure ageneral-purpose computer, a special-purpose computer, or aspecial-purpose processing device to perform a certain function or groupof functions. Portions of computer resources used can be accessible overa network. The computer-executable instructions may be, for example,binaries, intermediate format instructions such as assembly language,firmware, or source code. Examples of computer-readable media that maybe used to store instructions, information to be used, and/orinformation created during methods according to described examplesinclude magnetic or optical disks, flash memory, USB devices providedwith non-volatile memory, networked storage devices, and so on.

Devices implementing methods according to this disclosure can comprisehardware, firmware, and/or software and can take any of a variety ofform factors. Typical examples of such form factors include laptops,smart phones, small-form-factor personal computers, personal digitalassistants, and so on. Functionality described herein also can beembodied in peripherals or add-in cards. Such functionality also can beimplemented on a circuit board among different chips or differentprocesses executing in a single device, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthis disclosure.

Although a variety of examples and other information have been used toexplain various aspects within the scope of the appended claims, nolimitation of the claims should be implied based on particular featuresor arrangements in such examples, as one of ordinary skill would be ableto use these examples to derive a wide variety of implementations.Furthermore, and although some subject matter may have been described inlanguage specific to examples of structural features and/or methodsteps, it should be understood that the subject matter defined in theappended claims is not necessarily limited to those described featuresor acts. For example, functionality of the various components can bedistributed differently or performed in components other than thoseidentified herein. Therefore, the described features and steps aredisclosed as examples of components of systems and methods that aredeemed to be within the scope of the following claims.

What is claimed is:
 1. A method implemented on a handheld communicationdevice having a processor, an image capture device, and a display, themethod comprising: identifying a route from a first geographic positionof the handheld communication device to a point of interest; capturing afirst image at the first geographic position of the handheldcommunication device using the image capture device; presenting, on thedisplay, the captured first image and the route from the firstgeographic position of the handheld communication device to the point ofinterest; determining a first indicator for a first direction to travelalong the route to the point of interest based on map data describingstreets, buildings, and other points of interest within a field of viewof the captured first image; and visually augmenting the captured firstimage with the first indicator of the first direction to travel alongthe route to the point of interest, wherein the first indicator of thefirst direction to travel is overlaid on the captured first image. 2.The method according to claim 1, further comprising: capturing a secondimage at a second geographic position of the handheld communicationdevice along the route; determining a second indicator for a seconddirection to travel along the route from the second geographic positionto the point of interest; and visually augmenting the captured secondimage with the second indicator of the second direction to travel alongthe route, wherein the second indicator of the second direction totravel is overlaid on the captured second image.
 3. The method accordingto claim 1, further comprising: receiving a search input for the pointof interest; and identifying the point of interest received in thesearch input, wherein the identified point of interest is obstructed byan in-screen object.
 4. The method according to claim 1, wherein theidentified route from the first geographic position of the handheldcommunication device to the point of interest is a pedestrian route. 5.The method according to claim 1, wherein the first indicator comprisesturn arrows that are overlaid onto the captured first image to indicateupcoming turns in the route as a user travels along the route.
 6. Themethod according to claim 1, wherein travel directions are displayedwith the first indicator of the first direction to travel.
 7. The methodaccording to claim 1, wherein street names are displayed with the firstindicator of the first direction to travel.
 8. The method according toclaim 1, further comprising: in response to determining that there areno streets, buildings, and other points of interest within the field ofview of the captured first image, displaying a top-down view of a map onthe handheld communication device.
 9. The method according to claim 2,further comprising updating the captured first image presented on thedisplay to the captured second image in response to a user panning thehandheld communication device.
 10. The method according to claim 1,wherein the captured first image comprises a real-time live video feed.11. The method according to claim 3, wherein the search input is asearch term entered in a search field by a user.
 12. A non-transitorycomputer-readable medium storing instructions which, when executed byone or more processors, cause the one or more processors to: identify aroute from a first geographic position of a handheld communicationdevice to a point of interest; capture, by an image capture device ofthe handheld communication device, a first image at the first geographicposition of the handheld communication device using the image capturedevice; present, on a display of the handheld communication device, thecaptured first image and the route from the first geographic position ofthe handheld communication device to the point of interest; determine afirst indicator for a first direction to travel along the route to thepoint of interest based on map data describing streets, buildings, andother points of interest within a field of view of the captured firstimage; and visually augment the captured first image with the firstindicator of the first direction to travel along the route to the pointof interest, wherein the first indicator of the first direction totravel is overlaid on the captured first image.
 13. Thecomputer-readable medium according to claim 12, further comprising:capturing, by the image capture device of the handheld communicationdevice, a second image at a second geographic position of the handheldcommunication device along the route; determining a second indicator fora second direction to travel along the route from the second geographicposition to the point of interest; and visually augmenting the capturedsecond image with the second indicator of the second direction to travelalong the route, wherein the second indicator of the second direction totravel is overlaid on the captured second image.
 14. Thecomputer-readable medium according to claim 12, further comprising:receiving a search input for the point of interest; and identifying thepoint of interest received in the search input, wherein the identifiedpoint of interest is obstructed by an in-screen object.
 15. Thecomputer-readable medium according to claim 12, wherein the identifiedroute from the first geographic position of the handheld communicationdevice to the point of interest is a pedestrian route.
 16. Thecomputer-readable medium according to claim 12, wherein the firstindicator comprises turn arrows that are overlaid onto the capturedfirst image to indicate upcoming turns in the route as a user travelsalong the route.
 17. A system comprising: a memory; and one or moreprocessors configured to perform a method comprising: identifying aroute from a first geographic position of a handheld communicationdevice to a point of interest; capturing, by an image capture device ofthe handheld communication device, a first image at the first geographicposition of the handheld communication device using the image capturedevice; presenting, on a display of the handheld communication device,the captured first image and the route from the first geographicposition of the handheld communication device to the point of interest;determining a first indicator for a first direction to travel along theroute to the point of interest based on map data describing streets,buildings, and other points of interest within a field of view of thecaptured first image; and visually augmenting the captured first imagewith the first indicator of the first direction to travel along theroute to the point of interest, wherein the first indicator of the firstdirection to travel is overlaid on the captured first image.
 18. Thesystem according to claim 17, further comprising: capturing, by theimage capture device of the handheld communication device, a secondimage at a second geographic position of the handheld communicationdevice along the route; determining a second indicator for a seconddirection to travel along the route from the second geographic positionto the point of interest; and visually augmenting the captured secondimage with the second indicator of the second direction to travel alongthe route, wherein the second indicator of the second direction totravel is overlaid on the captured second image.
 19. The systemaccording to claim 17, further comprising: receiving a search input forthe point of interest; and identifying the point of interest received inthe search input, wherein the identified point of interest is obstructedby an in-screen object.
 20. The system according to claim 17, whereinthe identified route from the first geographic position of the handheldcommunication device to the point of interest is a pedestrian route.